High-performance Monitoring Architecture for Large-scale Distributed Systems Using Event Filtering

Monitoring is an essential process to observe and improve the reliability and the performance of large-scale distributed (LSD) systems. In an LSD environment, a large number of events is generated by the system components during its execution or interaction with external objects (e.g. users or processes). Monitoring such events is necessary for observing the run-time behavior of LSD systems and providing status information required for debugging, tuning and managing such applications. However, correlated events are generated concurrently and could be distributed in various locations in the applications environment which complicates the management decisions process and thereby makes monitoring LSD systems an intricate task. In this paper, we present a scalable high-performance monitoring architecture for LSD systems using an efficient event filtering mechanism to detect and classify interesting local and global events and disseminate the monitoring information to the corresponding endpoints management applications (such as debugging and reactive control tools). Our architecture also supports dynamic and flexible reconfiguration of the monitoring mechanism via its instrumentation and subscription components. The intrusiveness of the monitoring process is minimized by reducing the event traffic and distributing the monitoring load. In this paper, we describe and motivate the monitoring approach and the components design of the monitoring system that we are developing to observe the run-time behavior of LSD systems and improve their reliability and performance.